Skip to content

HADOOP-16591 Fix S3A ITest*MRjob failures.#1487

Closed
sidseth wants to merge 1 commit intoapache:trunkfrom
sidseth:HADOOP-16591
Closed

HADOOP-16591 Fix S3A ITest*MRjob failures.#1487
sidseth wants to merge 1 commit intoapache:trunkfrom
sidseth:HADOOP-16591

Conversation

@sidseth
Copy link
Contributor

@sidseth sidseth commented Sep 20, 2019

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@sidseth
Copy link
Contributor Author

sidseth commented Sep 20, 2019

Test command run: mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=12 -Ds3guard -Dauth -Ddynamo
Region: us-east-2
There's the usual flaky test failures. Otherwise, all tests succeed.

@hadoop-yetus
Copy link

hadoop-yetus commented Sep 20, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 79 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 1189 trunk passed
+1 compile 31 trunk passed
+1 checkstyle 24 trunk passed
+1 mvnsite 37 trunk passed
-1 shadedclient 98 branch has errors when building and testing our client artifacts.
+1 javadoc 24 trunk passed
0 spotbugs 60 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 58 trunk passed
_ Patch Compile Tests _
+1 mvninstall 32 the patch passed
+1 compile 26 the patch passed
+1 javac 26 the patch passed
+1 checkstyle 17 the patch passed
+1 mvnsite 30 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
-1 shadedclient 27 patch has errors when building and testing our client artifacts.
+1 javadoc 20 the patch passed
+1 findbugs 60 the patch passed
_ Other Tests _
+1 unit 78 hadoop-aws in the patch passed.
+1 asflicense 23 The patch does not generate ASF License warnings.
1927
Subsystem Report/Notes
Docker Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1487/1/artifact/out/Dockerfile
GITHUB PR #1487
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 732c1198d1c9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3f223be
Default Java 1.8.0_222
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1487/1/testReport/
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1487/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Sep 21, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 110 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 1455 trunk passed
+1 compile 37 trunk passed
+1 checkstyle 28 trunk passed
+1 mvnsite 42 trunk passed
-1 shadedclient 116 branch has errors when building and testing our client artifacts.
+1 javadoc 34 trunk passed
0 spotbugs 67 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 65 trunk passed
_ Patch Compile Tests _
+1 mvninstall 38 the patch passed
+1 compile 28 the patch passed
+1 javac 28 the patch passed
+1 checkstyle 21 the patch passed
+1 mvnsite 37 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
-1 shadedclient 33 patch has errors when building and testing our client artifacts.
+1 javadoc 26 the patch passed
+1 findbugs 70 the patch passed
_ Other Tests _
+1 unit 92 hadoop-aws in the patch passed.
+1 asflicense 30 The patch does not generate ASF License warnings.
2355
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1487/1/artifact/out/Dockerfile
GITHUB PR #1487
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 314cce32d3ca 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d7d6ec8
Default Java 1.8.0_222
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1487/1/testReport/
Max. process+thread count 264 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1487/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@sidseth
Copy link
Contributor Author

sidseth commented Sep 21, 2019

The shadedclient failure is not related. More details at #1432 (comment)

@sidseth
Copy link
Contributor Author

sidseth commented Sep 21, 2019

/rerun

@sidseth
Copy link
Contributor Author

sidseth commented Sep 21, 2019

/rebuild

@steveloughran
Copy link
Contributor

steveloughran commented Sep 23, 2019

+1, committed to trunk. Thanks!

FWIW, in HADOOP-16570 I'm revisiting this change to see if it should also/should instead have the conf used to create the minicluster changed by forcing down the test properties by way of the relevant S3ATestUtils call. I'll leave this change here in though

@sidseth
Copy link
Contributor Author

sidseth commented Sep 24, 2019

Thanks. I wasn't sure about that change - especially if the test configs set fs.defaultFS. OTOH, the MiniDFS cluster will likely overwrite that setting anyway - so should be safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran Awaiting requested review from steveloughran

@bgaborg bgaborg Awaiting requested review from bgaborg

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sidseth @hadoop-yetus @steveloughran